PSYC5141: Experimental Methods for Psychology

Mark Hurlstone, Lancaster University

Week 4, Lecture 2

Non-Parametric Within-Participants ANOVA

Lecture Overview

• Introduction to the Friedman test

  • When and why we might use Friedman

  • How it differs from Kruskal–Wallis

• Calculating the Friedman test statistic

  • Ranking observations within participants

  • Calculating and interpreting the test statistic

  • Measuring effect size using Kendall's W

• Follow-up comparisons

  • Wilcoxon signed-rank tests

  • Planned comparisons, post-hoc tests and multiple-comparison corrections

• Summary

  • Friedman versus parametric within-participants ANOVA

  • Key distinctions between the non-parametric procedures

Introducing the Friedman Test

From Kruskal–Wallis to Friedman

Both tests use ranks, but the way ranks are assigned reflects the experimental design

When might Friedman be appropriate?

• Parametric repeated-measures ANOVA models differences between condition means

• Its assumptions include independence between participants, normally distributed errors and sphericity

• When the data are ordinal or the assumptions of parametric analysis are poorly supported, a rank-based alternative may be useful

• The Friedman test does not require normality or sphericity

Introducing the Friedman test

The Friedman test is a non-parametric alternative to single-factor within-participants ANOVA

• Appropriate for three or more related conditions

• Each participant provides an observation in every condition

• Observations are ranked separately within each participant

• Tests whether the conditions differ systematically in their rankings

• Produces a test statistic that can be evaluated using a chi-squared approximation

A Within-Participants Experiment

• A psychologist investigates whether misinformation warnings improve people's ability to distinguish between true and false information on social media

• Independent variable: Type of misinformation warning, with three levels:
  • \(A_1\): No warning
  • \(A_2\): Generic warning
  • \(A_3\): Specific warning
• Dependent variable: Percentage of social media posts correctly classified as true or false
• Design: all five participants complete all three conditions

Example data

The logic of Friedman

• Suppose each participant receives ranks of 1, 2 and 3 according to their accuracy in the three conditions

• If warning condition has no systematic effect, we wouldn't expect one condition to consistently receive higher ranks than the others

• However, if specific warnings improve performance, we'd expect the specific-warning condition to receive relatively high ranks across participants

The Friedman test evaluates how consistently the conditions receive different rankings

Understanding the ranking procedure

Step 1: Rank observations within each participant

• Rank each participant's scores separately, from lowest (1) to highest (3)

• Each participant contributes one rank to each condition

• The ranks within each participant always sum to 6 (assuming three conditions, including when ties receive average ranks)

• Unlike Kruskal–Wallis, we do not pool scores across participants before ranking

Step 2: Calculate the rank totals for each condition

• Each condition has five ranks, and the total of all rank sums is 30
• The expected rank total for each condition under the null hypothesis is:

\[ \frac{n(k+1)}{2}=\frac{5(4)}{2}=10 \]

What do the rank totals tell us?

• If warning condition has no systematic influence, the conditions should have similar rank totals

• The expected rank total for each condition is 10

• Our observed rank totals are 6, 9 and 15

• The specific-warning condition consistently receives the highest rank

Main question: Are the observed differences in rank totals sufficiently large to reject the null hypothesis?

Introducing the Friedman formula

Symbols:

• \(n\) = number of participants
• \(k\) = number of conditions
• \(R_j\) = sum of ranks for condition \(j\)

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Introducing the Friedman formula

Symbols:

• \(n\) = number of participants
• \(k\) = number of conditions
• \(R_j\) = sum of ranks for condition \(j\)

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Square each condition's rank total and sum across conditions

Introducing the Friedman formula

Symbols:

• \(n\) = number of participants
• \(k\) = number of conditions
• \(R_j\) = sum of ranks for condition \(j\)

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Adjusts the calculation for the number of participants and conditions

Introducing the Friedman formula

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Introducing the Friedman formula

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Introducing the Friedman formula

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Introducing the Friedman formula

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Introducing the Friedman formula

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Introducing the Friedman formula

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Introducing the Friedman formula

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Introducing the Friedman formula

For our example:

• \(n = 5\)
• \(k = 3\)
• \(R_1 = 6,\ R_2 = 9,\ R_3 = 15\)

Interpreting the omnibus result

Chi-squared critical values (χ²)

Conclusion: The Friedman test is statistically significant, χ²(2) = 8.40, p < .05. The warning conditions differ systematically in their rankings.

Effect size: Kendall's W

Kendall's \(W\) is a measure of effect size for the Friedman test:

• Measures the degree of agreement in how participants rank the conditions
• Ranges from 0 (no overall agreement) to 1 (complete agreement)
• Larger values indicate greater consistency in condition rankings across participants

Symbols:

• \(Q\) = Friedman test statistic (\(8.40\))
• \(n\) = number of participants (\(5\))
• \(k\) = number of conditions (\(3\))

For our example:

Guidelines for interpreting Kendall's W

• These are descriptive guides rather than established cut-offs
• For our example, W = .84 indicates strong agreement in how participants ranked the three warning conditions.

Follow-up comparisons

Why do we need follow-up comparisons?

• The Friedman test is an omnibus test: it evaluates whether conditions differ systematically in their rankings

• A significant Friedman result does not identify which conditions differ

• As with parametric within-participants ANOVA, we can investigate specific differences using planned comparisons or post-hoc tests

Planned comparisons: Wilcoxon signed-rank test

• The Wilcoxon signed-rank test is a non-parametric alternative to the paired-samples t-test

  • Compares two related conditions using paired observations

  • Calculates the difference between each participant's two scores

  • Ranks the absolute differences, excluding zero differences

  • Uses the signs of the differences to assess whether there is a systematic tendency for one condition to produce higher scores

• Example: We might predict that participants receiving a specific warning (A₃) will achieve higher accuracy than when receiving no warning (A₁)

Post-hoc comparisons

• Following a significant Friedman test, we may want to investigate all possible pairwise differences

  • Pairwise Wilcoxon signed-rank tests are a common post-hoc procedure

  • Each test compares scores from two related conditions

  • Adjustments for multiple comparisons help control the familywise error rate

• ​As in the previous lecture, we can use Holm-adjusted p-values

Interpreting follow-up comparisons

The Friedman omnibus test was statistically significant:

\[ \chi^2(2)=8.40,\quad p=.015 \]

However, the pairwise Wilcoxon signed-rank tests, using exact two-sided p-values with Holm adjustment, produced the following results:

A significant omnibus test does not guarantee significant follow-up comparisons.​

Within-Participants ANOVA versus Friedman

Summary

• The Friedman test is a non-parametric alternative to single-factor within-participants ANOVA

• It ranks observations within each participant, rather than across the entire dataset

• The Friedman statistic assesses whether the conditions differ systematically in their rankings

• Wilcoxon signed-rank tests can be used to investigate specific differences between related conditions, with appropriate adjustments for multiple comparisons

That's all folks!